Skip to content

feat: sensitivity_from_table() -- post-hoc Sobol/Morris from a table (#113) - #118

Merged
jc-macdonald merged 4 commits into
mainfrom
feat/113-sensitivity-from-table
Aug 19, 2026
Merged

feat: sensitivity_from_table() -- post-hoc Sobol/Morris from a table (#113)#118
jc-macdonald merged 4 commits into
mainfrom
feat/113-sensitivity-from-table

Conversation

@jc-macdonald

Copy link
Copy Markdown
Collaborator

Stacked on #117

This PR is stacked on feat/114-surrogate-accuracy (#117) -- it uses fit_surrogate()'s new cv_r2 (from #114) as a trust signal for the sensitivity result. Base is set to that branch rather than main; retarget to main once #117 merges (or I'll do it).

Summary

screen() needs a live, re-runnable simulator and a proper Saltelli/Morris sample design -- neither of which an arbitrary already-collected ResultsTable has, so its indices can't be computed retroactively from whatever points happen to already be in a table. That rules out the literal ask in #113 ("reuse whatever Sobol machinery already backs screen()" applied directly to stored data) -- but there's a standard, sound alternative: fit a cheap surrogate over the table, then run screen()'s existing machinery against the surrogate's predict() instead of a fresh, expensive simulator call. Since the surrogate is cheap to query, you get proper variance-based indices without any new evaluations.

Why this actually fixes the motivating case

#113 was filed because VBPCApy's ARD prior strength (hp_va) has a mechanistic U-shaped effect (too weak = no pruning, too strong = over-pruning) that a marginal Spearman correlation shows as near-zero, while a co-varying, purely-monotonic factor (xprobe_fraction) that had nothing to do with the believed-dominant effect showed up as the strongest correlate. The new test test_sobol_detects_nonmonotonic_effect reproduces exactly this shape (y = (a - 0.5)**2 + 0.1*b, a U-shaped and symmetric, b linear) and confirms Sobol S1 through the surrogate correctly ranks a as dominant (importance > 0.5 vs b's remainder) -- the case a marginal correlation would get backwards.

Design

  • sensitivity_from_table(results, factors, *, method="sobol", surrogate_method="rf", ...): fits fit_surrogate() on the continuous subset of factors, then calls screen() with the surrogate's .predict as run_fn.
  • Only continuous factors are screened (matches screen()'s own contract exactly); non-continuous keys elsewhere in the table's configs are simply ignored by the encoder rather than causing a mismatch.
  • Returns TableSensitivity(importance, surrogate_cv_r2) -- surrogate_cv_r2 (fit_regime_surrogate / RegimeSurrogate should report held-out or OOB accuracy #114) is included directly in the result so callers don't have to separately fit and check accuracy before trusting the indices for a given observable.
  • Fails fast with screen()'s own "at least one continuous factor" message if none remain, rather than letting a degenerate empty-feature surrogate fit produce a confusing sklearn error.

Test plan

  • just ci green locally (ruff format/lint, mypy --strict, coverage 99.51%, 340 passed -- 8 new)
  • test_sobol_detects_nonmonotonic_effect: reproduces the exact U-shaped-effect failure mode from Post-hoc Sobol/Morris total-effect sensitivity from an existing ResultsTable #113 and confirms it's fixed
  • Morris method also runs end-to-end
  • Non-continuous factors in the input list are dropped, not an error
  • Empty continuous-factor list raises with screen()'s own message
  • A surrogate fit to pure noise triggers the forwarded fit_surrogate low-accuracy warning

…113)

screen() needs a live, re-runnable simulator and a proper Saltelli/
Morris sample design -- neither of which an arbitrary already-collected
ResultsTable has, so its indices can't be computed retroactively from
whatever points happen to be in the table. sensitivity_from_table()
bridges the gap: fit a cheap surrogate over the table (fit_surrogate,
#114), then run screen()'s existing Sobol/Morris machinery against the
surrogate's predict() instead of a fresh, expensive evaluation.

This directly fixes the motivating failure mode: a symmetric U-shaped
factor effect (e.g. VBPCApy's ARD prior strength -- too weak under-
prunes, too strong over-prunes, optimum in the middle) has near-zero
marginal Spearman correlation despite dominating the variance. Sobol
S1 through the surrogate correctly ranks it above a co-varying linear
factor in the new test (test_sobol_detects_nonmonotonic_effect).

Only continuous factors are screened, matching screen()'s own
contract; the surrogate is fit on that same continuous subset so
non-continuous keys elsewhere in the table's configs are ignored
rather than causing an encoding mismatch. TableSensitivity.
surrogate_cv_r2 carries the fitted surrogate's cross-validated
accuracy (#114) so callers can judge whether to trust a given
observable's indices before acting on them.

Closes #113
Base automatically changed from feat/114-surrogate-accuracy to main August 19, 2026 14:38
@jc-macdonald
jc-macdonald merged commit e32a75d into main Aug 19, 2026
4 checks passed
@jc-macdonald
jc-macdonald deleted the feat/113-sensitivity-from-table branch August 19, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant